PATENT APPLICATION 



IN THE U.S. PATENT AND TRADEMARK OFFICE 

m, 

Express Mail Label No.: EL 482 000 575 US d| 
Attorney Docket No.: R & G C-312 *i 
November 28, 2000 ^ 

Box Patent Application 

Assistant Commissioner for Patents 

Washington, DC 20231 

Sir: 

Transmitted herewith for filing is the [X] utility [ ] design 
patent application of: 

Inventor (s) : Robert Edward Meredith SWANN and 
Robert Wei Liang TAN 



For : IMAGE PROCESSING 

Pursuant to 37 CFR 1.53, the above-identified application is 
being submitted without the filing fee , and it is respectfully 
requested that this application be granted a serial number and 
filing date and that the undersigned be notified of the date 
by which the filing fee must be submitted. Enclosed are: 

[X] Specification 

[ ] Declaration or Oath 

[X] 5 Drawing Sheets [ ] Formal 

[X] Informal 

[ ] Amendment Before First Office Action 

[ ] Information Disclosure Statement 

[ ] Assignment, with Recordation Form Cover Sheet. A check 

for $ is enclosed to cover the recording fee. 
[X] Acknowledgment Postal Card 

[X] Priority is claimed under 35 USC 119 based on United 

Kingdom Application No. 0016222.2 , filed June 30, 2000 . 
[ ] A certified copy of the priority application is 
enclosed. 

[X] Pursuant to 37 CFR 1.53, this application is being filed 
without the declaration or oath. Please notify the 
undersigned of the due date for submitting the 
declaration or oath. A Power of Attorney in favor of the 



Patent Application Transmittal - Page 2 

undersigned will be submitted later. Please direct all 
telephone calls to the undersigned at telephone number 
(616) 381-1156. Please address all correspondence to: 



The inventors are Dr. Robert Edward Meredith SWANN and 
Robert Wei Liang TAN ; residing at Cambridge and Herts, 
Great Britain ; having the following post office 
addresses: 13 Honey Hill Mews Pound Hill Cambridge CB3 
OAL, Great Britain and 37 Orchard Road Melbourn Royston 
Herts SG8 6HH^ Great Britain ; and having the following 
citizenship: Great Britain . 
[ I Pursuant to 37 CFR 1.52, the enclosed application is in 
the form of a foreign language text: 

[ ] An English translation and a statement that the 

English translation is accurate are enclosed. 
[ ] Please notify the undersigned of the due date for 



FLYNN, THIEL, BOUTELL & TANIS, P.C. 
2026 Rambling Road 
Kalamazoo, MI 49008-1699 



submitting an English translation and a statement 
that the English translation is accurate. 

Respectfully submitted. 




David S. Goldenberg 



DSG/km 



FLYNN, THIEL, BOUTELL 



Dale H. Thiel Reg. No. 24 323 

David G. Boutell Reg. No. 25 072 

Ronald J. Tanis Reg. No. 22 724 

Terryence F. Chapman Reg. No. 32 549 

Mark L. Maki Reg. No. 36 589 

David S. Goldenberg Reg. No. 31 257 

Sidney B. Williams, Jr. Reg. No. 24 949 

Liane L. Churney Reg. No. 40 694 

Brian R. Tumm Reg. No. 36 328 



& TANIS, P.C. 



202 6 Rambling Road 



Kalamazoo, MI 49008-1699 



Phone: (616) 381-1156 
Fax: (616) 381-5465 



End : 



Listed above 



PATENT APPLICATION 



"Express Mail" Mailing Label No.: EL 482 000 575 US 

Date of Mailing: November 28, 2000 

Applicant (s) : Robert Edward Meredith SWANN et al 

Title: IMAGE PROCESSING 

Serial No. : Unknown 

Filed: Unknown 

Atty Docket No.: R & G C-312 



Assistant Commissioner for Patents 
Washington DC 20231 

EXPRESS MAILING CERTIFICATE 

Sir: 

I hereby certify that the attached paper or fee is being 
deposited with the United States Postal Service "Express Mail 
Post Office to Addressee" service under 37 CFR 1.10 on the 
date indicated above and is addressed to the Assistant 
CoDraiissioner for Patents, Washington DC 20231. 

FLYNN, THIEL, BOUTELL & TANIS, P.C. 

By: (}Lfi, f^laipir 

Date: November 28, 2000 

Document (s) attached: Patent Application Transmittal 

Specification, Claims & Abstract 
5 Drawing Sheets 
Postal Card 

Telephone: (616) 381-1156 
191. 9912 



Express Mailing Label #EL 482 000 575 US 



IMAGE PROCESSING 

This invention relates to image processing and is of 
particular benefit to clustering data in binary images. 

5 In a number of image processing applications there is a 

requirement to cluster objects together, i.e. objects 
which are related due to their close proximity to each 
other. For example in document image processing, text 
objects that are close to each other may be clustered 

10 together to foirm paragraphs. With binary image data this 

clustering can be performed by merging, i.e. removing gaps 
between objects that are less than a specified distance 
limit. The difficulty in merging binary objects into 
clusters is the determination of the distance limit. If 

15 the distance limit is too small then some related objects 

remain separate. If the distance limit is too large then 
gaps between clusters may be removed incorrectly. 

If the separation between clusters is significantly larger 
than the gaps between objects within a cluster, then the 

20 setting of the distance limit is easy. In the cases where 
the separation between clusters is similar to the gaps 
between objects in a cluster, deciding the distance limit 
for merging is more difficult. If the separation^between 
clusters is smaller than the gaps between" objects in a 

25 cluster then it may be impossible for a single distance 

limit to merge objects into clusters without also 
incorrectly joining clusters together. In the example of 
text proces'sing of document images, -merging text into 
paragraphs is easy if the separation of the paragraphs is 

30 significantly larger than the gap between letters. 

However if the gap between letters is less than the 
separation between paragraphs (as is often the case in a 
document with many fonts and sizes of text) , then it may 



not be possible to cluster all the text into paragraphs 
successfully using a simple binary merging operation. 

In image processing applications where the clustering of 
binary objects is difficult because of the close proximity 
of the clusters, it is often helpful to use additional 
information to segment the binary image. The information 
used to segment the binary image is generally more useful 
if taken from a separate source or earlier form of the 
image. In the example of text processing in document 
images, the binary image of the text objects may be 
segmented according to background colour, calculated from 
the original greyscale image of the document. 
Unfortunately segmentation of an image can be difficult 
and many techniques do not adequately account for slowly 
varying features or incomplete region boundaries. 

We have appreciated that the process of segmenting and 
merging in order to cluster objects in a binary image may 
be made more successful and computationally efficient if 
they are combined into a single process, where the 
segmentation information is represented as the boundaries 
between regions. Accordingly a preferred embodiment of 
the invention clusters together objects in a binary image 
by removing the gaps between objects in the cases where 
the gaps are less than a specified distance limit and do 
not cross a region boundary. 

We have observed that the merging of objects in a omary 
image into clusters can benefit from segmentation of that 
image. If the segmentation can separate clusters without 
dissecting them then it reduces the likelihood of 
incorrectly merging clusters together. This can simplify 
the requirements of the merging operation making it easier 
to successfully set a distance limit for the merging- We 
have also observed that the requirement of merging objects 



in a binary image into clusters also simplifies the task 
of segmentation. As an isolated task the segmentation 
would need to be able to separate the whole image into 
distinct regions. However the merging operation has a 
distance limit which will keep clusters that are well 
separated isolated. Thus the demands on the segmentation 
are reduced to separating regions where the clusters would 
otherwise be merged together. The benefit of the 
invention performing both the merging and segmentation 
simultaneously is to take advantage of the reduced 
requirements of the segmentation information and the 
simplification in the setting of the merging distance 
limit . 

The invention is defined with more precision in the 
appended claims to which reference should now be made. 

A preferred embodiment of the invention will now be 
described in detail by way of an example with reference to 
the accompanying drawings in which: 

Figpare 1 shows schematically apparatus for identifying 
addresses on complex envelopes; 

Figure 2 shows schematically a conventional method for 
applying both segmentation and merging to a binary^ image; 
Fignre 3 shows schematically an embodiment of the 

invention used to perform the tasks of both segmentation 
and merging on a binary image; 

Figure 4 shows benefit of using regions boundaries for 
segmentation m the case of segmenting text in a document 
image according to background colour; and 
Figure 5 shows schematically apparatus for identifying 
addresses on complex envelopes using an embodiment of the 



In an example of identifying addresses on complex 
envelopes in a mail processing system there is a 
requirement to cluster text objects into paragraphs that 
may be addresses. As with many other image processing 
5 applications the merging of binary text objects into 
paragraphs can benefit from the use of additional 
information to segment the image. Accordingly in the case 
of processing dociament images such as mail we have 
proposed in our British Patent Application No . [ ] 

10 filed on the same day as the current application, a method 
of clustering related text objects in an image document . 
The aforementioned patent uses two segmentations, one 
according to text colour and one according to background 
colour, to segment the binary image of the text objects. 

15 In the example shown Figure 1 the apparatus to identify 

addresses on complex envelopes, uses a simplified version 
that creates a segmentation from the background 
information only. 

In Figure 1, a scanner 2 produces a 256 level greyscale 
20 image from the input document. It could of course produce 
other resolutions of greyscale image or indeed a colour 
image. This greyscale image then passes to a text object 
extraction unit 4 which identifies the text objects in the 
image and produces a binary image of these. 

25 At the same time, the greyscale image passes to a global 
information segmentation unit 6 which creates a 
segmentation for the image based on the background 
greyscale level by defining regions where the grey level 
is the same or within a predetermined range. This could 

30 also use colour information. This segmentation data 
relating to background grey level is then passed to a 
segmentation unit 8 which also receives the binary image 
from the text object extraction unit 4. This then 



segments the binary image of text according to the global 
background information supplied to it. 

The output data for this is then passed to the merging 
unit 10 which, for each segmented region, merges text 
objects according to the distance between letters. 
Letters which are separated by less than a predetermined 
distance are merged together and ones which are further 
apart are not. This produces text blocks which are passed 
to a sorting unit 12. This sorts the text blocks 
according to those most likely to contain an address (in 
the case of a mail processing system) and passes these to 
an optical character recognition (OCR) unit 14. 

Steps 8 and 10 in Figure 1 are a binary segmentation and 
merger. A conventional approach to both segmentation and 
merging of a binary image is shown in Figure 2. This 
requires the segmentation to separate the image into 
complete regions. The merging operation can then be 
applied to each region separately. In Figure 2 a 
segmentation map is created from segmentation information 
by a segmentation mapping unit 16. This map of segmented 
regions is combined with the binary image (of text or 
other objects) in a binary image segmentation unit 18. 
This outputs, in turn, separated regions of the binary 
image to a merging unit 20 which removes gaps between 
objects below a specified distance limit with each region. 
The output of this is then a sequence of clustered binary 
objects which do not cross region boundaries. The 
clusters are constrained by the distance limit of the 
merging applied. The task of using the segmentation 
information to separate the image into distinct regions is 
difficult to do accurately and therefore this system does 
have drawbacks. 



An embodiment of the invention as shown in Figure 3 is an 
alternative to this. Instead of attempting to use the 
segmentation information to separate the image into 
complete regions, the task is reduced to extracting the 
region boundaries. Removing gaps between objects in the 
binary image then performs the merging operation. 
However, gaps are only removed if they are less than the 
specified distance limit and do not cross a region 
boundary. The result is effectively a simultaneous 
segmentation and merging operation with the advantage that 
at no point does the image have to be completely 
segmented. In fact, since the region boundaries do not 
have to be complete, the process is significantly simpler 
than forming an isolated segmentation. 

The system of Figure 3 comprises a boundary extraction 
unit 22 which extracts region boundaries from segmentation 
information derived from the input greyscale image or some 
other source. This segmentation information may be 
information defining changes in background colour or 
greyscale which exceeds a predetermined limit. This gives 
a binary image of region boundaries which forms one of the 
inputs to the merging and segmentation unit 24. The other 
input to this is the binary image, which in the case of 
mail 'processing would come from the text object extraction 
unit. 4 of Figure 1. 

The segmentation and merging unit then merges together 
objects in the binary image without crossing any region 
boundaries. Merging is performed by removing gaps between 
objects that are below the merging distance limit and 
which do not cross a region boundary. Normally, such a 
process will be performed by conventional hardware 
programmed with software to perform this process. 
However, dedicated hardware circuitry could be provided to 
implement the segmentation and merging. The benefit of 



using boundaries for segmentation is illustrated in Figure 
4a) -e). Figure 4a) and b) show the original and 
background to a document in a mail processing system. In 
order to aid the clustering of text objects in the 
paragraphs. The background has been extracted from the 
original image to segment the binary text objets. This 
segmentation provided by the background is shown in Figure 
4b. As can be seen, the background areas are obviously 
distinct and there is no difficulty in separating the 
image into complete distinct regions according to 
background. 

Figures 4c) and d) show original and background of another 
envelope. The top half of the image is a table of figures 
and the bottom half has some text. However, the text on 
the lower section is printed on a background which darkens 
from right to left. This gradient makes it difficult to 
segment the image into distinct regions. On the left 
side, it is clear that the image has upper and lower parts 
which should be separated. On the right there is no 
apparent boundary. Complete segmentation would therefore 
be very difficult. However, region boundaries are simple 
to extract from the background and give a good accurate 
representation of where merging can and cannot occur. 
Thus, the region boundaries can be extracted from, the 
background information and these are illustrated in Figure 
4e) . In this, the line dividing upper and lower parts is 
distinct in the left-hand portion of the image but 
gradually decreases in distinctness towards the right-hand 
edge. Using the system of Figure 3, no merging can take 
place across this boundary. 

A further advantage of this simultaneous merging and 
segmentation is that, whereas normal segmentation 
information needs to be able to segment the whole image, 
in this particular system, it only needs to represent a 



region boundary. This can be merely a Ixne. It does not 
need to enclose the distinct region. When used with 
document image processing, the text objects, the 
background colour, text colour, text orientation, etc., 
can all be used to segment the whole image. However, with 
the current technique, incomplete boundaries such as bold 
lines, location of images and logos, etc., can all be used 
successfully to aid clustering of text objects. In 
addition, repetitive segmentations are normally 
computationally intensive. The present technique requires 
only a binary image of the lines not to be crossed during 
merging. Thus, the multiple segmentations represent a 
case of Or-ing a number of binary images to create a 
complete binary image of the region boundaries. This is 
computationally much less intensive. 

The embodiment shown in Figure 5 shows the merging and 
segmentation system of Figure 3 used in the apparatus of 
Figure 1 and replacing the segmentation unit 8 and the 
merging unit 10. The segmentation information input to 
the boundary extraction units 22 comes from the global 
segmentation unit 6. The binary image input to the 
merging and segmentation unit comes from the text object 
extraction unit 4. The output of the segmentation and 
merging unit 24 is a set of text objects grouped into text 
blocks- These are then sorted for the text block most 
likely to be the address by the sorting unit 12 before 
passing to the OCR 14. The system can easily be 
implemented to use text colour segmentation instead of 
greyscale segmentation. 

The invention performs the clustering of objects in a 
binary image where the clusters are described by a maximum 
distance allowed between objects and by some information 
that implies segmentation between clusters. The 
segmentation information is supplied as a binary image of 



region boundary Lines that are not crossed during the 
clustering operation. The region boundaries do not have 
to be complete. Since the invention is a general image 
processing technique for clustering of binary objects 
5 there are numerous applications. The main example used in 
this description has been the clustering of text objects 
into paragraphs in document images. Other applications 
could include: 



Biological Image Processing 
10 E.g. Clustering binary image of foreign bodies 

infecting cells, where the segmentation region 
boundaries are the cell walls. 



Geographical Image Processing 

E.g. Clustering live stock in images of fields where 
15 the region boundaries are the hedge rows. 



Image Processing in military applications 
E.g. Clustering of personnel in a building, where the 
binary image of personnel locations may be taken from 
a thermal imaging system and the segmentation region 
20 boundaries are taken as the walls in the building 

plans . 
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CLAIMS: 

1. An image processing system for producing 
clusters for related objects for subsequent analysis 
comprising, means for supplying a multi-level digital 
5 representation of the image, means for identifying 

predetermined objects in the image and supplying data 
defining their locations, means for deriving boundary data 
from the multi-level digital representation, the boundary 
data representing boundaries between different regions of 
10 the image, means for clustering the predetermined objects 
into groups in dependence on their proximity to each other 
and in dependence on the boundary data, and means for 
supplying data relating to the groups for subsequent 
analysis . 

15 2 . An image processing system according to claim 1 

in which the boundary data is derived from another 
representation of the image. 

3. An image processing system according to claim 1 
in which the boundary data is derived from a separate 

20 source containing data relating to the image. 

4. An image processing system according to. claim 1 
in which the boundary data is derived from background data 
from the multi-level digital representation. 

5. An image processing system according to claim 
25 4 in which the background data is colour data. 

6. An image processing system according to claim 
4 in which the background data is greyscale data. 

7. An image processing system according to any 
preceding claim in which the means for clustering objects 



together clusters objects which are separated by less than 
a predetermined limit. 

8. An image processing system according to any 
preceding claim in which the means for clustering objects 
together clusters objects which are not separated by any 
boundary defined by the boundary data. 

9. A method for processing images for producing 
clusters of related objects for subsequent analysis 
comprising the steps of, supplying a multi-level digital 
representation of the image, identifying predetermined 
objects in the image, supplying data defining the 
locations of the predetermined objects, deriving boundary 
data from the multi-level digital representation, the 
boundary data representing boundaries between regions of 
the original image, and clustering the predetermined 
objects into groups in dependence on their proximity to 
each other and in dependence on the boundary data. 

10. A method according to claim 9 in which the 
boundary data is derived from another representation of 
the image . 

11. A method according to claim 9 in which the 

boundary data is derived from a separate source containing 
data relating to the image. 

12. A method according to claim 7 in which the 
boundary data is derived from background data in the 
multi-level digital representation. 

13. A method according to claim 12 in which the 
background data is colour data. 



14. A method according to claim 12 in which the 
background data is grayscale data. 

15. A method according to any of claims 10 to 14 in 
which the step of clustering objects together clusters 
objects separated by less than a predetermined limit. 

16- A method according to any of claims 10 to 15 in 
which the step of clustering objects together clusters 
objects which are not separated by any boundary defined by 
the boundary data. 



ABSTRACT : 



An image processing system produces clusters of 
related objects for subsequent analysis. It receives a 
multi-level digital representation of the image and 
5 predetermined objects are identified by extraction units 

(4) . A segmentation unit (6) drives boundary data from 
the multi-level digital representation, the boundary data 
representing boundaries between different regions of the 
image. A clustering unit (8) clusters the predetermined 
10 objects into groups in dependence on their proximity to 

each other and in dependence on the boundary data and 
supplies this cluster data for subsequent analysis. 
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Figure 2 
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Figure 3 
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